The Body as Musical Instrument

This chapter explores the possibility of thinking of the human body as musical instrument. It builds on the philosophy of phenomenology to discuss body schemata that might be considered “instrumental” and discusses the diversity of bodies proposed by body theory to consider the incorporation of digital technology. Concepts of embodied interaction from the scientific field of human–computer interaction are discussed with an eye toward musical application. The history of gestural musical instruments is presented, from the Theremin to instruments from the STEIM studio. The text then focuses on the use of physiological signals to create music, from historical works of Lucier and Rosenboom to recent performances by the authors. The body as musical instrument is discussed in a dynamic of coadaptation between performer and instrument in different configurations of body and technology

Prototyping audiovisual performance tools: A hackathon approach

We present a user-centered approach for prototyping tools for performance with procedural sound and graphics, based on a hackathon. We also present the resulting prototypes. These prototypes respond to a challenge originating from earlier stages of the research: to combine ease-of-use with expressiveness and visibility of interaction in tools for audiovisual performance. We aimed to convert sketches, resulting from an earlier brainstorming session, into functional prototypes in a short period of time. The outcomes include open-source software base released online. The conclusions reflect on the methodology adopted and the effectiveness of the prototypes

AVUI: Designing a toolkit for audiovisual interfaces

The combined use of sound and image has a rich history, from audiovisual artworks to research exploring the potential of data visualization and sonification. However, we lack standard tools or guidelines for audiovisual (AV) interaction design, particularly for live performance. We propose the AVUI (AudioVisual User Interface), where sound and image are used together in a cohesive way in the interface; and an enabling technology, the ofxAVUI toolkit. AVUI guidelines and ofxAVUI were developed in a three-stage process, together with AV producers: 1) participatory design activities; 2) prototype development; 3) encapsulation of prototype as a plug-in, evaluation, and roll out. Best practices identified include: reconfigurable interfaces and mappings; object-oriented packaging of AV and UI; diverse sound visualization; flexible media manipulation and management. The toolkit and a mobile app developed using it have been released as open-source. Guidelines and toolkit demonstrate the potential of AVUI and offer designers a convenient framework for AV interaction design

Creative Applications of Interactive Mobile Music

This chapter examines the foundational research concerning the applications of interactive mobile music conducted at Sony Computer Science Laboratory Paris and at Culture Lab Newcastle. It analyzes the forms and formats that music can take on when deployed on mobile devices and wireless infrastructures and looks at the development of conceptual thinking of mobile music creation outside the sphere of consumer markets and commercial applications. The chapter also discusses efforts to leverage the possibilities of contextual sensing coupled with dynamic media delivery systems to create new musical experiences that can be shared by groups of performers and listeners

BioMuse to Bondage: Corporeal Interaction in Performance and Exhibition

This chapter looks at technologies of biosensing and techniques of sonification that invoke physicalization of sound to create a sense of intimacy in musical performance and gallery exhibition. By comparing the dynamic across stage and gallery settings, we will see how the corporeal activation of sound can become a key not just to decode experimental performance, but as an entry point to possible intimate spaces created by digital interaction. We will draw upon notions of physicalization, enaction, and embodiment to describe how a visceral, nearly carnal situation can be created in technology mediated work

The Sound of Photographic Image

This text presents three works produced in the period 1998–2004 that are based on the sonification of photographic images. Two of the works, 9m14s Over Vietnam, and Bondage.rmx, were compositions for recorded CD release, while the third, Bondage, was an interactive installation. All three utilized similar processes to translate visual information to sound, and to allow musical composition with imagery

Mapping Out Instruments, Affordances, and Mobiles

This paper reviews and extends questions of the scope of an interactive musical instrument and mapping strategies for expressive performance. We apply notions of embodiment and affordance to characterize gestural instruments. We note that the democratization of sensor technology in consumer devices has extended the cultural contexts for interaction. We revisit questions of mapping drawing upon the theory of affordances to consider mapping and instrument together. This is applied to recent work by the author and his collaborators in the development of instruments based on mobile devices designed for specific performance situations

User-centered design of a tool for interactive computer-generated audiovisuals

The present study aims to design a tool for interactive computer-generated audiovisuals. In this paper, we investigate if the tools for audio- visual performance and composition have caught up with the growing interest and the practices in the field. We have adopted a user-centered design approach for our study, based on interviews and a workshop with practitioners. The interviews identified key themes – expressivity, ease of use and connection with the audience – that were explored in the workshop. During the workshop, a novel methodology was adopt- ed – reboot – which expands upon the bootlegging technique. Key ide- as regarding audiovisual performance gathered from the interviews; sketches for novel audiovisual tools resulting from the workshop; and the reboot technique, are the main contributions of this study

Back To The Cross-Modal Object: A Look Back At Early Audiovisual Performance Through The Lens Of Objecthood

This paper looks at 2 early digital audiovisual performance works, solo work Overbow and the group Sensors_Sonics_Sights (S.S.S) and describes the compositional and performance strategies behind each one. We draw upon the concept of audiovisual objecthood proposed by Kubovy and Schutz to think about the different ways in which linkages between vision and audition can be established, and how audio-visual objects can be composed from the specific attributes of auditory and visual perception. The model is used as a means to analyze these live audio-visual works performed using sensor-based instruments. The fact that gesture is not the only visual component in these performances, and is the common source articulating sound and visual output, extends the classical 2-way audiovisual object into a three-way relationship between gesture, sound, and image, fulfilling a potential of cross-modal objects